Autonomous driving control apparatus and program product

ABSTRACT

In an autonomous driving control apparatus, a drive control unit determines, based on first and second images respectively captured by first and second cameras and an autonomous driving condition, a value of at least one controlled variable for autonomous driving of a vehicle, and outputs, to a vehicle control unit, the value of the at least one controlled variable to thereby cause the vehicle control unit to carry out a task of autonomously driving the vehicle. A camera monitor unit determines whether at least one of the first and second cameras has malfunctioned. The camera monitor unit limits, based on the at least one of the first and second directional regions corresponding to the at least of the first and second cameras, the autonomous driving condition when determining that the at least one of the first and second cameras has malfunctioned.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is based on and claims the benefit of priority fromJapanese Patent Application 2018-104690 filed on May 31, 2018, thedisclosure of which is incorporated in its entirety herein by reference.

TECHNICAL FIELD

The present disclosure relates to autonomous driving control apparatusesand program products. More particularly, the present disclosure relatesto these apparatuses and programs, each of which is capable ofautonomously driving a vehicle based on images of a forward region ofthe vehicle in its travelling direction.

BACKGROUND

A known autonomous driving control apparatus includes an autonomousdriving system installed in a host vehicle for autonomously driving thehost vehicle when the host vehicle is set to an autonomous driving mode.In contrast, the driver's manual driving of the host vehicle is carriedout when the host vehicle is set to a manual driving mode.

SUMMARY

According to a first exemplary aspect of the present disclosure, thereis provided an autonomous driving control apparatus installable in avehicle. The autonomous driving control apparatus includes a cameramonitor unit configured to determine whether at least one of first andsecond cameras has malfunctioned, and limit, when determining that oneof the first and second cameras has malfunctioned, an autonomous drivingcondition based on one of first and second directional regionscorresponding to one of the first and second cameras havingmalfunctioned. A drive control unit of the apparatus is configured todetermine the value of at least one controlled variable in accordancewith the limited autonomous driving condition to thereby cause thevehicle control unit to continuously carry out a task of autonomouslydriving the vehicle in accordance with the determined value of the atleast one controlled variable.

According to a second exemplary aspect of the present disclosure, thereis provided a program product for a vehicle. The computer programproduct includes a non-transitory computer-readable storage medium, anda set of computer program instructions stored in the computer-readablestorage medium. The instructions cause a computer to carry out

1. A first step of determining whether one of first and second camerashas malfunctioned

2. A second step of limiting, when it is determined one of the first andsecond cameras has malfunctioned, an autonomous driving condition basedon one of first and second directional regions corresponding to one ofthe first and second cameras has malfunctioned

BRIEF DESCRIPTION OF THE DRAWINGS

Other aspects of the present disclosure will become apparent from thefollowing description of embodiments with reference to the accompanyingdrawings in which:

FIG. 1 is a block diagram schematically illustrating an example of theoverall structure of an autonomous driving control system according toan exemplary embodiment of the present disclosure;

FIG. 2 is a perspective view schematically illustrating an example ofthe outer appearance of a camera module illustrated in FIG. 1;

FIG. 3 is a longitudinal cross-sectional view of the camera module forshowing an example of the internal structure of the camera module;

FIG. 4 is a view schematically illustrating an example of therelationship among

(1) A set of an angular field of view and a depth of field of awide-angle camera of the camera module

(2) A set of an angular field of view and a depth of field of anarrow-angle camera of the camera module

(3) A set of an angular field of view and a depth of field of atelephoto camera of the camera module; and

FIG. 5 is a flowchart schematically illustrating a camera monitorroutine according to the exemplary embodiment of the present disclosure.

DETAILED DESCRIPTION OF EMBODIMENT Inventor's Viewpoint

A known autonomous driving control apparatus, which is for exampledisclosed in Japanese Patent Application Publication No. 2017-157067,includes an autonomous driving system installed in a host vehicle forautonomously driving the host vehicle when the host vehicle is set to anautonomous driving mode. In contrast, the driver's manual driving of thehost vehicle is carried out when the host vehicle is set to a manualdriving mode.

The published autonomous driving system installed in a host vehicleincludes an autonomous driving system, which is comprised of anindividual computer and is capable of controlling autonomous driving ofthe host vehicle in level 3 of the previously defined six autonomousdriving levels.

The published autonomous driving control apparatus also includes apreventive safety system, which is comprised of an individual computerand is capable of controlling autonomous driving of the host vehicle inlevel 2 of the previously defined levels. The autonomous driving systemcontrols all of the acceleration, steering, and braking of the hostvehicle in level 3 of the six autonomous driving levels. The predictivesafety system controls some of the acceleration, steering, and brakingof the host vehicle in level 2 of the six autonomous driving levels.

Specifically, the autonomous driving control apparatus disclosed in thepublished patent document is configured such that the predictive safetysystem performs autonomous driving of the host vehicle in the level 2 ofthe six autonomous driving levels in response to an autonomous request.Thereafter, the autonomous driving control apparatus is configured suchthat the autonomous driving system performs autonomous driving of thehost vehicle in the level 3 of the six autonomous driving levels when itis determined that a predetermined level 3 authorization condition issatisfied.

Each of the predictive safety system and autonomous driving system isconfigured to perform an emergency limp-home mode to pull over the hostvehicle to a safe place upon it being determined that there is amalfunction in the other of the predictive safety system and autonomousdriving system. While performing the emergency limp-home mode, each ofthe predictive safety system and autonomous driving system is configuredto shift autonomous driving of the host vehicle to level 1 of the sixautonomous driving levels in response to a driver's intervention to thedriving of the host vehicle. Note that level 1 of the six autonomousdriving levels represents a driver assistance mode for assisting thedriver's acceleration, steering, and braking of the host vehicle.

The published autonomous driving control apparatus described in thepublished patent document is configured as a redundant vehicle controlsystem, i.e. a duplicated vehicle control system, comprised of theindividual computer of the autonomous driving system, and the individualcomputer of the predictive safety system. This redundant vehicle controlsystem of the autonomous driving control apparatus enables autonomousdriving of the host vehicle to be continuously carried out even if oneof the he autonomous driving system and the predictive safety system hasmalfunctioned. Additionally, the published autonomous driving controlapparatus comprised of the level 3 autonomous driving system and thelevel 2 predictive safety system results in a lower cost than autonomousdriving control apparatuses each with duplicated level 3 autonomoussystems.

A malfunction in one of the predictive safety system and autonomousdriving system of the published autonomous driving control apparatuscauses the other of the predictive safety system and autonomous drivingsystem to perform the emergency limp-home mode to pull the host vehicleover to a safe place. This may complicate continuous execution ofautonomous driving of the host vehicle.

Let the published autonomous driving control apparatus be configured todetermine a travelling route of the host vehicle using images capturedby an in-vehicle camera, and to perform autonomous steering of the hostvehicle. In this case, a malfunction in the in-vehicle camera may makeit difficult for the published autonomous driving control apparatus topull over the host vehicle to a safe place even if the apparatusincludes the duplicated vehicle control system.

Such a malfunction of the in-vehicle camera may therefore require theoperation mode of the host vehicle to be switched from the autonomousdriving mode to the manual driving mode. This may make it difficult forthe driver to perform autonomous driving of the host vehicle using thepublished autonomous driving control apparatus, resulting in thepublished autonomous driving control apparatus having a lower usability.

In view of the above circumstances, a first aspect of the presentdisclosure seeks to provide autonomous driving control apparatuses, eachof which is configured to perform autonomous driving of a host vehicleusing an image captured by an in-vehicle camera, and is capable ofcontinuously performing autonomous driving of the host vehicle even ifthere is a malfunction in the in-vehicle camera.

A second aspect of the present disclosure seeks to provide programproducts, each of which causes a processor to perform autonomous drivingof a host vehicle using an image captured by an in-vehicle camera, andis capable of causing a processor to continuously perform autonomousdriving of the host vehicle even if there is a malfunction in thein-vehicle camera.

According to a first exemplary aspect of the present disclosure, thereis provided an autonomous driving control apparatus installable in avehicle that includes: at least first and second cameras configured torespectively capture at least first and second images of at least firstand second directional regions that are at least partly different fromeach other around the vehicle; and a vehicle control unit configured tocontrol a travelling state of the vehicle. The autonomous drivingcontrol apparatus includes a drive control unit configured to determine,based on the first and second images and an autonomous drivingcondition, a value of at least one controlled variable for autonomousdriving of the vehicle, and output, to the vehicle control unit, thevalue of the at least one controlled variable to thereby cause thevehicle control unit to carry out a task of autonomously driving thevehicle. The autonomous driving control apparatus includes a cameramonitor unit configured to determine whether at least one of the firstand second cameras has malfunctioned, and limit, when determining thatone of the first and second cameras has malfunctioned, the autonomousdriving condition based on one of the first and second directionalregions corresponding to one of the first and second cameras havingmalfunctioned. The drive control unit is configured to determine thevalue of the at least one controlled variable in accordance with thelimited autonomous driving condition to thereby cause the vehiclecontrol unit to continuously carry out the task of autonomously drivingthe vehicle in accordance with the determined value of the at least onecontrolled variable.

According to a second exemplary aspect of the present disclosure, thereis provided a program product for a vehicle that includes at least firstand second cameras configured to respectively capture at least first andsecond images of at least first and second directional regions that areat least partly different from each other from the vehicle. The vehicleincludes a vehicle control unit configured to control a travelling stateof the vehicle, and a drive control unit. The drive control unit isconfigured to determine, based on the first and second images and anautonomous driving condition, a value of at least one controlledvariable for autonomous driving of the vehicle, and output, to thevehicle control unit, the value of the at least one controlled variableto thereby cause the vehicle control unit to carry out a task ofautonomously driving the vehicle. The computer program product includesa non-transitory computer-readable storage medium, and a set of computerprogram instructions stored in the computer-readable storage medium. Theinstructions cause a computer to carry out

1. A first step of determining whether one of the first and secondcameras has malfunctioned

2. A second step of limiting, when it is determined one of the first andsecond cameras has malfunctioned, the autonomous driving condition basedon one of the first and second directional regions corresponding to oneof the first and second cameras has malfunctioned

This configuration of each of the first and second exemplary aspectstherefore enables autonomous driving of the vehicle to be continuouslycarried out based on the limited autonomous driving conditions even ifone of the first and second cameras has malfunctioned. This thereforeeliminates the need to discontinue, i.e. cancel, the autonomous drivingof the vehicle, thus providing the autonomous driving apparatus beingmore convenient for drives of the vehicle

Embodiment

The following describes an exemplary embodiment of the presentdisclosure with reference to the accompanying drawings.

The following describes an example of the configuration of an autonomousdriving system 1 according to the exemplary embodiment of the presentdisclosure with reference to FIG. 1.

Referring to FIG. 1, the autonomous driving system 1 is for exampleinstalled in an autonomous vehicle V, such as a passenger vehicle, andconfigured to autonomously drive the autonomous vehicle V, which will bereferred to simply as a vehicle V.

The autonomous driving system 1 for example includes an autonomousdriving control electronic control unit (ECU) 10 as its main component.In addition to the autonomous driving control ECU 10, which is simplyreferred to as an ECU 10, the autonomous driving system 1 includes, forexample, an own-vehicle position sensor unit 12, a surrounding situationsensor unit 13, a communication module 14, a road information storage16, an output unit 17, an input unit 18, and a camera module 30. Thesecomponents 12 to 18 and 30 are communicably connectable to the ECU 10.

In the vehicle V, a navigation system 22, a drive power control ECU 24,a brake power control ECU 26, and a steering control ECU 28 areinstalled. These components 20 to 28 are communicably connectable toeach other via an in-vehicle network 20 installed in the vehicle V.

The own-vehicle position sensor unit 12 is capable of measuring thecurrent position of the vehicle V. For example, the own-vehicle positionsensor unit 12 includes a global positioning system (GPS) receiver and agyro sensor. The GPS receiver is configured to receive, via a GPSantenna, GPS signals, which are sent from GPS satellites, to therebycalculate for example the latitude and longitude of the position of theGPS antenna of the vehicle V based on the received GPS signals asposition data of the GPS antenna.

The gyro sensor is configured to measure a value of the angular velocityaround at least one of predetermined pitch axis, roll axis, and yaw axisof the vehicle V. These pitch, roll, and yaw axes pass through thecenter of gravity of the vehicle V. The pitch axis represents ahorizontal axis parallel to the width direction of the vehicle V, theyaw axis represents a vertical axis parallel to the height direction ofthe vehicle V, and the roll axis represents a longitudinal axis parallelto the longitudinal direction of the vehicle V.

The own-vehicle position sensor unit 12 is configured to calculate, as acurrent position of the vehicle V, a current position of the center ofthe gravity of the vehicle V based on a predetermined positionalrelationship between the GPS antenna and the center of gravity of thevehicle V, the measured position data of the GPS antenna, and themeasured value of the angular velocity around at least one of the pitch,roll, and yaw axes of the vehicle V. Then, the own-vehicle positionsensor unit 12 is configured to send a measurement signal indicative ofthe current position of the vehicle V to the ECU 10.

The surrounding situation sensor unit 13 is capable of measuring acurrent surrounding situation around the vehicle V; the currentsurrounding situation around the vehicle V can be used by the ECU 10.

For example, the surrounding situation sensor unit 13 includes forexample at least one of a laser radar sensor, a millimeter-wave sensor,a ultrasonic-wave sensor. The surrounding situation sensor unit 13 isconfigured to

(1) Transmit, to a predetermined surrounding region around the vehicleV, probing waves

(2) Receive reflection waves, i.e. echoes, generated based on reflectionof the transmitted probing waves by objects located around the vehicle V

(3) Detect, based on the received reflection waves, the existence,location, size, and/or distance of each of the objects

For example, the objects include

1. One or more surrounding travelling vehicles travelling around thevehicle V

2. One or more surrounding obstacles that is located around the vehicleV and obstruct travelling of the vehicle V

The one or more surrounding travelling vehicles include

1. A preceding vehicle travelling on the same lane of the vehicle V andlocated in front of the vehicle V

2. A preceding vehicle travelling on an adjacent lane of the lane of thevehicle V and located at the front side of the vehicle V

3. One or more on coming vehicles

4. One or more incoming vehicles

The one or more surrounding obstacles include

1. One or more stopped vehicles

2. One or more fallen objects

3. One or more stopped objects

4. One or more pedestrians

The surrounding situation sensor unit 13 is capable of sending, to theECU 10, a measurement signal indicative of the current surroundingsituation.

The communication module 14 enables the ECU 10 to communicate, by radio,with traffic servers established outside the vehicle V to thereby obtain

(1) Traffic condition information

(2) Weather condition information

The weather condition information represents the weather condition, suchas a bright condition, a rain condition, a cloud condition, a snowcondition, a fog condition, or a sandstorm condition around the vehicleV, which can be collected by at least one of the traffic servers.

The traffic condition information includes various types of trafficinformation about each road on which the vehicle V can travel. Forexample, the traffic condition information can include a speed limit ofeach travelable road, information about whether passing is permitted foreach travelable road, information about whether there are trafficregulations for each travelable road.

The road information storage 16, which is for example comprised of arewritable storage medium, such as a flash ROM, rewritably stores roadinformation about one or more roads on which the vehicle V is scheduledto travel; the road information about each of the scheduled roadsincludes

1. The type of the road whether the road is an urban road or anexpressway, how many lanes the road has, and whether there are one ormore oncoming lanes in the road

2. The width of each lane in the road

3. The center line of each lane in the road

4. The curvature of the road when the road is a curved road

5. The positions of one or more stop lines marked on the road

6. The positions of one or more traffic lights when the one or moretraffic lights are provided on the road

Note that the navigation system 22 described later is for exampleconfigured to provide the road information to the ECU 10.

The output unit 17 includes, for example, an image display and a soundspeaker, and is capable of visibly and/or audibly outputting, to adriver of the vehicle V, various messages using the image display and/orthe sound speaker.

The input unit 18 includes, for example, operation switches and/oroperation levers, and enables a driver of the vehicle V to input, to theECU 10, various instructions for autonomous driving of the vehicle Vusing the operation switches and/or operation levers.

The camera module 30 is capable of capturing images from the surroundingregion around the vehicle V.

For example, the camera module 30 is comprised of three cameras withrespective different angular fields of view, i.e. a wide-angle camera 32w, a narrow-angle camera 32 n whose angular field of view narrower thanthat of the wide-angle camera 32 w, and a telephoto camera 32 t whoseangular field of view is narrower than that of the narrow-angle camera32 n.

Note that the angular field of view of each of the lenses 33 w, 33 n,and 33 t is at least one of

1. A diagonal angular field of view corresponding to a diagonaldirection of a captured image of the corresponding camera, i.e. adiagonal line of the light-receiving surface of the correspondingimaging device

2. A horizontal angular field of view corresponding to a horizontaldirection of a captured image of the corresponding camera, i.e. ahorizontal direction of the light-receiving surface of the correspondingimaging device

3. A vertical angular field of view corresponding to a verticaldirection of a captured image of the corresponding camera, i.e. avertical direction of the light-receiving surface of the correspondingimaging device.

This exemplary embodiment uses the horizontal angular field of view ofeach lens 33 w, 33 n, 33 t as the angular field of view of thecorresponding lens.

Images, such as two-dimensional frame images, captured by the respectivecameras 32 w, 32 n, and 32 t are used for recognizing lane markers on ascheduled road on which the vehicle V is scheduled to travel, and forrecognizing objects existing in the surrounding region around thevehicle V. The structure of the camera module 30 according to theexemplary embodiment will be described in detail later.

The navigation system 22 is configured to store various road informationitems and map information items about many roads that the vehicle V istravelable.

Specifically, the navigation system 22 is configured to continuouslydisplay a road map on the image display of the output unit 17 around thecurrent position of the vehicle V, and display, on the road map, aselected route from the current position of the vehicle V to adestination in response to when a driver of the vehicle V inputs thedestination using the input unit 17. Then, the navigation system 22 isconfigured to obtain the road information about one or more roadsincluded in the selected route, and provide the road information aboutthe one or more scheduled roads to the ECU 10. The navigation system 22is also configured to provide, to a driver of the vehicle V, a visibleand audible guide for enabling the vehicle V to travel along theselected route using the image display and the sound speaker.

The drive power control ECU 24 is configured to control at least onedrive actuator, such as an internal combustion engine and/or a motor;the at least one drive actuator outputs controlled drive power tothereby rotatably drive driving wheels of the vehicle V.

Specifically, the drive power control ECU 24 is configured to controlthe at least one drive actuator to thereby cause the at least one driveactuator to generate drive power based on an operated amount of anaccelerator pedal operated by a driver of the vehicle V when theoperation mode of the vehicle V is set to a manual driving mode, thusrotatably driving the driving wheels based on the generated drive power.

The drive power control ECU 24 is also configured to receive requesteddrive power from the ECU 10 when the operation mode of the vehicle V isset to an autonomous driving mode, and control the at least one driveactuator to thereby cause the at least one drive actuator to generatedrive power that satisfies the requested drive power, thus rotatablydriving the driving wheels based on the generated drive power.

Note that the operation mode of the vehicle V can be changed by, forexample, the ECU 10 in accordance with an instruction sent by a driver'soperation of the input unit 17.

The brake power control ECU 26 is configured to control at least onebrake actuator; the at least one brake actuator outputs controlled brakepower to thereby brake the vehicle V.

Specifically, the brake power control ECU 26 is configured to controlthe at least one brake actuator to thereby cause the at least one brakeactuator to generate brake power based on an operated amount of a brakepedal operated by a driver of the vehicle V when the operation mode ofthe vehicle V is set to the manual driving mode, thus slowing down thevehicle V based on the generated brake power.

The brake power control ECU 26 is also configured to receive requestedbrake power from the ECU 10 when the operation mode of the vehicle V isset to the autonomous driving mode, and control the at least one brakeactuator to thereby cause the at least one brake actuator to generatebrake power that satisfies the requested brake power, thus slowing downthe vehicle V based on the generated brake power.

The steering control ECU 28 is configured to control a motor included ina steering mechanism of the vehicle V; the motor of the steeringmechanism outputs controlled steering torque that steers the steeringwheel of the vehicle V.

Specifically, the steering control ECU 28 is configured to control themotor of the steering mechanism to thereby cause the motor to generate,as the steering torque, controlled assist torque based on an operatedamount of the steering wheel operated by a driver of the vehicle V whenthe operation mode of the vehicle V is set to the manual driving mode.This controlled assist torque assists the driver's steering operation ofthe steering wheel.

The steering control ECU 28 is also configured to receive a requestedsteering angle from the ECU 10 when the operation mode of the vehicle Vis set to the autonomous driving mode, and control the motor to therebycause the motor to generate the steering torque that satisfies therequested steering angle.

Next, the following describes an example of the configuration of the ECU10.

Referring to FIG. 1, the ECU 10 serves as, for example, an autonomousdriving control apparatus, and is comprised of, for example, aprocessing unit, such as a central processing unit (CPU) 2. The ECU 10can be comprised of another type of processing unit, such as anapplication specific integrated circuit (ASIC). The ECU 10 is alsocomprised of a memory 4 including, for example, a non-transitorytangible storage media that include, for example, a random access memory(RAM) and a read only memory (ROM).

Various programs including control programs for causing the CPU 2 toperform various tasks, i.e. routines, are stored in the memory 4. Inaddition, various data items usable by the CPU 2 are also stored in thememory 4. The CPU 2 reads at least one of the control programs from thememory 4, and executes the at least one control program to therebyexecute the routine corresponding to the at least one control program.In other words, the CPU 2 executes the at least one control program tothereby implement predetermined functional modules, such as a drivecontrol unit 6 and a camera monitor unit 8 (see dashed blocks in FIG.1), based on the at least one control program. In addition, the CPU 2 isconfigured to control overall operations of the ECU 10.

The drive control unit 6 executes an autonomous driving control taskthat causes at least one of the ECUs 24, 26, and 28 to autonomouslydrive the vehicle V when the operation mode of the vehicle V is set tothe autonomous driving mode.

Specifically, the ECU 10 sets the operation mode of the vehicle V to theautonomous driving mode when an instruction indicative of autonomousdriving is input from a driver of the vehicle V through the input unit18.

The instruction indicative of the autonomous driving includes, forexample, execution information about the autonomous driving to beexecuted; the execution information includes automatic cruise control(ACC), autonomous steering of the vehicle V, and/or autonomous drivingof the vehicle V in a predetermined level of the previously definedlevels.

The ACC is configured to adjust the speed of the vehicle V to therebycause the vehicle V to track a preceding vehicle in front of the vehicleV. The autonomous steering is configured to autonomously control thesteering wheel of the vehicle V to thereby cause the vehicle V to travelwithin the corresponding lane on each of one or more scheduled roads.

The autonomous driving is configured to autonomously drive the vehicle Vin the predetermined level of the previously defined levels instructedby the driver of the vehicle V.

Additionally, the drive control unit 6 obtains, from the navigationsystem 22,

(1) The driver's selected route from the current position of the vehicleV to the destination

(2) The road information about the one or more roads included in theselected route

Then, the drive control unit 6 sets, based on the execution informationabout the autonomous driving and the road information, autonomousdriving conditions required for at least one of the ECUs 24, 26, and 28to autonomous drive the vehicle V. The autonomous driving conditionsinclude, for example, a value of the speed of the vehicle V, a speedlimit, i.e. an upper speed limit, of the vehicle V during autonomousdriving, a value of the steering angle of the steering wheel, and a noautonomous-driving zone in one or more scheduled roads.

Additionally, the drive control unit 6 obtains the measurement signalsfrom the respective sensor units 12 and 13, obtains the images from therespective cameras 32 w, 32 n, and 32 t, and also obtains the trafficcondition information and weather condition information through thecommunication module 14. The road information set forth above, themeasurement signals, the images, the traffic condition information andweather condition information, which will be collectively referred to asautonomous-driving requirement information items, are obtained from thecorresponding devices.

Then, the drive control unit 6 determines a value of at least one ofcontrolled variables for the respective at least one drive actuator, theat least one brake actuator, and the motor of the steering mechanism inaccordance with the autonomous driving conditions and theautonomous-driving requirement information items each time receiving theautonomous-driving requirement information items.

Based on the calculated value of at least one of the controlledvariables for the respective at least one drive actuator, the at leastone brake actuator, and the motor of the steering mechanism, the drivecontrol unit 6 obtains at least one of

(1) Requested drive power for the at least one drive actuator

(2) Requested brake power for the at least one brake actuator

(3) Requested steering angle for the motor

Then, the drive control unit 6 outputs, to at least one of the drivepower control ECU 24, brake power control ECU 26, and steering controlECU 28, the corresponding at least one of the requested drive power forthe at least one drive actuator, requested brake power for the at leastone brake actuator, and requested steering angle. This causes at leastone of the drive power control ECU 24, brake power control ECU 26, andsteering control ECU 28 to execute a corresponding task of theautonomous driving instructed by the driver of the vehicle V.

Note that at least one of the ECUs 24, 26, and 28 serves as, forexample, a vehicle control unit configured to determine, based on frameimages captured by the respective cameras 32 w, 32 n, and 32 t and theautonomous driving conditions, a value of at least one controlledvariable that is needed to cause the vehicle V to autonomously travel.

The above autonomous driving control task of the drive control unit 6 iscontinuously, i.e. repeatedly, carried out while the operation mode ofthe vehicle V is set to the autonomous driving mode.

On the other hand, the camera monitor unit 8 is configured to

1. Monitor the operating state of each of the wide-angle camera 32 w,narrow-angle camera 32 n, and telephoto camera 32 t while the drivecontrol unit 6 is executing the autonomous driving control task setforth above

2. Determine whether one of the three cameras 32 w, 32 n, and 32 t hasmalfunctioned in accordance with the monitored result of each of thethree cameras 32 w, 32 n, and 32 t

3. Limit a part of the autonomous driving control task when determiningthat one of the three cameras 32 w, 32 n, and 32 t has malfunctioned toaccordingly enable the autonomous driving of the vehicle V to becontinuously carried out

Next, the following describes an example of the configuration of thecamera module 30, and an example of functions of the camera module 30with reference to FIGS. 2 and 3.

Referring to FIGS. 2 and 3, the camera module 30 is comprised of thethree cameras 32 w, 32 n, and 32 t, a camera casing 40, and a bracketassembly 50. The three cameras 32 w, 32 n, and 32 t are installed in thecamera casing 40.

Each of the cameras 32 w, 32 n, and 32 t includes, for example, acorresponding one of lens barrels 34 w, 34 n, and 34 t having opposingfirst and second opening ends, and a corresponding one of lenses 33 w,33 n, and 33 t coaxially located in the corresponding lens barrel to becloser to the first opening end of the corresponding lens barrel. Thisenables external light entering the lens barrel of each camera 32 w, 32n, and 32 t through the first opening end to be incident to thecorresponding lens 33 w, 33 n, 33 t.

Each of the cameras 32 w, 32 n, and 32 t also includes, for example, anunillustrated lens set located in the corresponding lens barrel to becloser to the second opening end of the corresponding lens barrel. Theunillustrated lens set for each camera 32 w, 32 n, and 32 t isconfigured to, for example, correct light, which has passed through thecorresponding lens 33 w, 33 n, 33 t, for optical aberration, such aschromatic aberration, of the corresponding lens 33 w, 33 n, 33 t.

Each of the cameras 32 w, 32 n, and 32 t also includes a correspondingone of imaging devices 35 w, 35 n, and 35 t, and a corresponding one ofrectangular plate-like imaging boards 36 w, 36 n, and 36 t.

Each of the imaging boards 36 w, 36 n, and 36 t has opposing first andsecond major surfaces, and each of the lens barrels 34 w, 34 n, and 34 tis mounted to the first major surface of the corresponding one of theimaging boards 36 w, 36 n, and 36 t. Each of the imaging devices 35 w,35 n, and 35 t is implemented on the first major surface of thecorresponding one of the imaging boards 36 w, 36 n, and 36 t.

Specifically, each of the lens barrels 34 w, 34 n, and 34 t is mountedat the periphery of its second opening end on the first major surface ofthe corresponding one of the imaging boards 36 w, 36 n, and 36 t suchthat the corresponding one of imaging devices 35 w, 35 n, and 35 t iscoaxial with the optical axis of the corresponding one of the lenses 33w, 33 n, and 33 t.

Each of the imaging devices 35 w, 35 n, and 35 t is for exampleconfigured by a color/monochrome charge-coupled device (CCD) imagesensor or a color/monochrome complementary metal oxide semiconductor(CMOS) image sensor. Each of the imaging devices 35 w, 35 n, and 35 t iscomprised of a plurality of light receiving elements, which respectivelycorrespond to a plurality of pixels, two-dimensionally arranged in botha vertical direction corresponding to the height direction of thevehicle V and a horizontal direction corresponding to the widthdirection of the vehicle V.

The light receiving elements of each imaging device 35 w, 35 n, 35 tconstitute a light receiving surface thereof, and the light receivingsurface of each imaging device 35 w, 35 n, 35 t is directed toward, forexample, the front end of the vehicle V. The lens 33 w, 33 n, 33 t ofeach camera 32 w, 32 n, 32 t is configured to focus light entering thecorresponding lens barrel 34 w, 34 n, 34 t on the light receivingsurface of the corresponding imaging device 35 w, 35 n, 35 t.

The camera casing 40 is comprised of a first casing segment 41 and asecond casing segment 42 that are assembled to constitute the cameracasing 40. For example, each of the first and second casing segments 41and 42 is made of a hard material with a relatively highheat-dissipation capacity, such as aluminum.

The first casing segment 41 has a substantially rectangular cup shape tohave a bottom wall 41 a, an opening wall 41 b opposite to the bottomwall 41 a, a first sidewall 41 c, and a second sidewall 41 d opposite tothe first opening sidewall 41 c. The first sidewall 41 c is comprised ofa peripheral edge and an opening defined around the peripheral edge. Theperipheral edge is comprised of a first end joined to a correspondingedge of the bottom wall 41 a, so that the first sidewall 41 c extends tobe perpendicular to the bottom wall 41 a. Similarly, the second sidewall41 d has a first end joined to a corresponding edge of the bottom wall41 a, so that the second sidewall 41 d extends to be perpendicular tothe bottom wall 41 a.

The first casing segment 41 has a flange 41 e extending away from asecond end of the first sidewall 41 a opposite to the first end thereofin perpendicular to the first sidewall 41 a.

The second casing segment 42 has a substantially rectangular dish shapeto have a bottom wall 42 a, an opening wall 42 b opposite to the bottomwall 42 a, a first sidewall 42 c, and a second sidewall 42 d opposite tothe first sidewall 42 c. The first sidewall 42 c has a first end joinedto a corresponding edge of the bottom wall 42 a, so that the firstsidewall 42 c extends to be perpendicular to the bottom wall 42 a.Similarly, the second sidewall 42 d has a first end joined to acorresponding edge of the bottom wall 42 a, so that the second sidewall42 d extends to be perpendicular to the bottom wall 42 a.

The second casing segment 42 is arranged to face the first casingsegment 41 such that a second end of each of the first and secondsidewalls 42 c and 42 d, which is opposite to the first end of thecorresponding sidewall, is joined to the second end of the correspondingone of the first and second sidewalls 41 c and 41 d of the first casingsegment 41 with, for example, bolts, thus constituting the camera casing40 in which an installation space is defined between the first andsecond casing segments 41 and 42.

The cameras 32 w, 32 n, and 32 t are fixedly mounted through the openingof the first sidewall 41 c of the first casing member 41 such that

(1) They are aligned in the vertical direction with clearancesthereamong

(2) The first opening ends of the respective lens barrels 34 w, 34 n,and 34 t are exposed from the first casing member 41

For example, as illustrated in FIGS. 2 and 3, the wide-angle camera 32 wis located at the lowermost portion of the first sidewall 41 c, thetelephoto camera 32 t is located at the uppermost portion of the firstsidewall 41 c, and the narrow-angle camera 32 n is located to be higherthan the wide-angle camera 32 w and lower than the telephoto camera 32t.

The camera casing 40 comprised of the first and second casing segments41 and 42 is arranged in a compartment of the vehicle V such that thebottom wall 41 a and the first sidewall 41 c of the first casing segment41 are located to be close to an inner surface 52 a of a frontwindshield 52 of the vehicle V while the opening wall 41 b of the firstcasing segment 41 is directed toward the lower direction of the vehicleV, and the extending direction of the flange 41 e is oriented toward,for example, the front end of the vehicle V.

The bracket assembly 50 is arranged to mount the camera casing 40, i.e.the camera module 30, to a portion of the inner surface 52 a of thefront windshield 52; the portion of the inner surface 52 a of the frontwindshield 52 has been determined to be out of the way of the driver'sview.

For example, the bracket assembly 50 is comprised of a bracket 54 andplural mounting pads 56.

The bracket 54 serves to be mounted to the inner surface 52 a of thefront windshield 52.

The bracket 54 has a substantially rectangular plate-like shape with asubstantially trapezoidal concave recess 54 a formed in a front side 54b thereof. The bracket 54 has opposing first and second major surfaces,and is arranged in the compartment of the vehicle V such that the firstmajor surface thereof is located alongside the inner surface 52 a of thefront windshield 52. The mounting pads 56 are distributedly arrangedbetween the inner surface 52 a of the front windshield 52 and the firstmajor surface of the bracket 54, so that the bracket 54 is fixedlymounted to the front windshield 52 through the mounting pads 56.

As described above, the cameras 32 w, 32 n, and 32 t are fixedly mountedthrough the opening of the first sidewall 41 c of the first casingmember 41 such that

(1) The telephoto camera 32 t, the narrow-angle camera 32 n, and thewide-angle camera 32 w are vertically aligned in this order from above

(2) The first opening ends of the respective lens barrels 34 w, 34 n,and 34 t are exposed from the first casing member 41

This results in, when the bracket 54, to which the camera module 30 ismounted, is attached to the inner surface 52 a of the front windshield52, optical axes At, An, and Aw of the respective cameras 32 t, 32 n,and 32 w

(1) Being aligned vertically in this order from above

(2) Extending toward the front end of the vehicle V

The concave recess 54 a is comprised of a centered inner-peripheral edgecorresponding to a shorter side of the trapezoidally-shaped concaverecess 54 a such that the first sidewall 41 c of the first casingsegment 41 is located below the centered inner-peripheral edge.

The concave recess 54 a is also comprised of a pair of oblique edgesrespectively extending obliquely outward from both ends of the centeredinner-peripheral edges toward, for example, the front end of the vehicleV.

The bracket assembly 50 is also comprised of, for example, a fixturewall 58 integrally extending from the centered inner-peripheral edge ofthe concave recess 54 a downward; the fixture wall 58, which hasopposing shorter top and bottom and opposing longer vertical sides, isdisposed in the opening of the first sidewall 41 c to fix the lensbarrels 34 w, 34 n, and 34 t to the peripheral wall of the firstsidewall 41 c, thus positioning the cameras 32 w, 32 n, and 32 t whiletheir lenses 33 w, 33 n, and 33 t are directed toward, for example, thefront of the vehicle V.

The bracket assembly 50 is further comprised of, for example, a pair ofoblique sidewalls 62 integrally extending from the respective obliqueedges of the concave recess 54 a downward. In other words, each of theoblique sidewalls 62, which has opposing top and bottom, is located suchthat the top of each of the oblique sidewalls 62 is joined to thecorresponding one of the oblique edges of the concave recess 54 a.

The oblique sidewalls 62 extend obliquely outward along the respectiveoblique edges of the concave recess 54 a from both the longer verticalsides of the fixture wall 58 toward, for example, the front end of thevehicle V while centering around the optical axes At, An, and Aw of therespective cameras 32 t, 32 n, and 32 w and being tapered.

Additionally, the bracket assembly 50 is comprised of, for example, abase wall 64 having a substantially trapezoidal shape that extendsobliquely outward from the bottom of the fixture wall 58 toward, forexample, the front of the vehicle V below the optical axes At, An, andAw of the respective cameras 32 t, 32 n, and 32 w. In other words, thebase wall 64 has opposing unparallel sides that are joined to thebottoms of the respective oblique sidewalls 62.

That is, the oblique sidewalls 62 and the base wall 64 joined theretosurround the optical axes At, An, and Aw of the respective cameras 32 t,32 n, and 32 w, resulting in the assembly of the oblique sidewalls 62and the base wall 64 constituting a hood 60 for preventing lightentering from outside of the walls 62 and 64.

This therefore enables light from the forward direction of vehicle V tobe incident to the cameras 32 w, 32 n, and 32 t via the front windshield52, so that the incident light is received by each of the cameras 32 w,32 n, and 32 t.

The light received by each of the cameras 32 w, 32 n, and 32 t isfocused by the corresponding lens 33 w, 33 n, 33 t and unillustratedlens set on the light receiving surface of the corresponding imagingdevice 35 w, 35 n, 35 t, so that each of the two-dimensionally arrangedlight-sensitive elements (pixels) receives a corresponding lightcomponent during a controllable shutter time, i.e. an exposure duration.Then, each of the light-sensitive elements converts a correspondingreceived light component into an electrical charge, i.e. an electricalsignal, corresponding to the intensity of the received light component,thus generating the electric signals as received light data, i.e.capturing a two-dimensional frame image.

In addition, the camera module 30 includes a control circuit board 44made of a rigid substrate, such as a glass epoxy substrate, and having asubstantially rectangular plate-like shape. The control circuit board 44is installed in the installation space defined between the first andsecond casing segments 41 and 42. The control circuit board 44 includesa control circuit 46 comprised of many electric and/or electronicelements.

Each of the cameras 32 w, 32 n, and 32 t additionally includes acorresponding one of imaging circuits 37 w, 37 n, and 37 t implementedto the corresponding one of the imaging boards 36 w, 36 n, and 36 t.

Each of the imaging circuits 37 w, 37 n, and 37 t is communicablyconnected to the corresponding one of the imaging devices 35 w, 35 n,and 35 t. Each of the imaging circuits 37 w, 37 n, and 37 t is alsocommunicably connected to the control circuit 46 implemented to thecontrol circuit board 44 via, for example, a corresponding one offlexible printed circuit boards (FPC) 38 w, 38 n, and 38 t.

Specifically, the control circuit 46 is configured to control thecorresponding imaging device 35 w, 35 n, and 35 t of each camera 32 w,32 n, 32 t in accordance with, for example, a predetermined value of theexposure duration, i.e. shutter time, and a predetermined value of aframe rate, cooperatively with the corresponding imaging circuit 37 w,37 n, 37 t to thereby cause the corresponding imaging device 35 w, 35 n,and 35 t to successively capture two-dimensional frame images from thesurrounding region around the vehicle V. Then, the control circuit 46receives, as image data items, the two-dimensional frame imagessuccessively captured by each of the cameras 32 w, 32 n, and 32 t.

The control circuit 46 obtains the image data items from each of thecameras 32 w, 32 n, and 32 t, and performs, based on the image dataitems, various tasks including an image recognition task that

(1) Recognizes lane markers on a scheduled road on which the vehicle Vis scheduled to travel

(2) Recognizes objects existing in the surrounding region around thevehicle V

Objects recognizable by the image recognition task can include, forexample, obstacles, such as pedestrians, bicycles, and other vehicles,and structures, such as traffic signals, traffic signs, and/orbuildings.

The camera module 30 includes at least one connector 48 mounted to thecontrol circuit board 44. The control circuit 46 is communicablyconnected to the ECU 10 via the at least one connector 48. The controlcircuit 46 is configured to output, to the ECU 10, the image data itemsobtained by each camera 32 w, 32 n, 32 t and the results of the imagerecognition task via the at least one connector 48 in response tocommands sent from the ECU 10.

In particular, referring to FIG. 4, each of the cameras 32 w, 32 n, and32 t has a corresponding one of imaging regions that are at least partlydifferent from each other.

Specifically, the lens 33 w and the imaging device 35 w of the camera 32w are configured to capture an image of a first imaging region definedaround the optical axis Aw, and the lens 33 n and the imaging device 35n of the camera 32 n are configured to capture an image of a secondimaging region defined around the optical axis An, which is at leastpartly different from the first imaging region. Similarly, the lens 33 tand the imaging device 35 t of the camera 32 t are configured to capturean image of a third imaging region defined around the optical axis At,which is at least partly different from the first and second imagingregions.

Each of the first to third imaging regions corresponds to, for example,a corresponding one of first to third directional regions, such asforward regions in the travelling direction of the vehicle V. Forexample, the lens 33 w of the wide-angle camera 32 w is comprised of awide-angle lens configured as, for example, a concave meniscus lens madeof, for example, a transparent material, such as glass. The lens 33 w isarranged such that its concave major surface is directed toward theimaging device 35 w.

The lens 33 n of the narrow-angle camera 32 n is comprised of, forexample, a narrow-angle lens having an angular field of view θn narrowerthan an angular field of view θw of the lens 33 w of the wide-anglecamera 32 w. The lens 33 n is configured as, for example, a concavemeniscus lens made of, for example, a transparent material, such asglass. The lens 33 n is arranged such that its concave major surface isdirected toward the imaging device 35 n.

Additionally, the lens 33 t of the telephoto camera 32 t is a comprisedof, for example, a telephoto lens having an angular field of view θtnarrower than the angular field of view θn of the lens 33 n of thenarrow-angle camera 32 n. The lens 33 t is configured as, for example, aconcave lens made of, for example, a transparent material, such asglass. The lens 33 t is arranged such that its concave major surface isdirected toward the front end of the vehicle V.

Because the wide-angle camera 32 w uses the wide-angle lens 33 w, theangular field of view θw of the lens 33 w of the wide-angle camera 32 wis set to a relatively wide angle of, for example, 120 degrees. Thewide-angle camera 32 w has a depth of field Dw within the angular fieldof view θw of the wide-angle lens 33 w; the depth of field Dw is set tobe within the range from a predetermined close point Dwc, which is, forexample, a point of the closest focus of the wide-angle lens 33 w, and apredetermined far point Dwf, which is, for example, a point of thefarthest focus of the wide-angle lens 33 w.

The lens barrel 34 n of the narrow-angle camera 32 n is positioned inthe first casing segment 41 such that at least a predetermined rearprincipal point of the narrow-angle lens 33 n is vertically andhorizontally aligned with a corresponding predetermined rear principalpoint of the wide-angle lens 33 w. Additionally, the optical axis An ofthe narrow-angle camera 32 n is eccentrically adjusted with respect tothe optical axis Aw of the wide-angle camera 32 w in the verticaldirection to thereby maintain the horizontal position of the opticalaxis An of the narrow-angle camera 32 n vertically aligning with thehorizontal position of the optical axis Aw.

Because the narrow-angle camera 32 n uses the narrow-angle lens 33 n,the angular field of view θn of the lens 33 n of the narrow-angle camera32 n is set to a middle angle narrower than the angular field of view θwof the lens 33 w, such as, for example, 60 degrees. These settingsenable the angular field of view θn of the lens 33 n of the narrow-anglecamera 32 n to be partially overlap with the angular field of view θw ofthe lens 33 w of the wide-angle camera 32 w in the normal directionperpendicular to the corresponding angular fields of view.

The narrow-angle camera 32 n has a depth of field Dn within the angularfield of view θn of the narrow-angle lens 33 n; the depth of field Dn isset to be within the range from a predetermined close point Dnc, whichis, for example, a point of the closest focus of the narrow-angle lens33 n, and a predetermined far point Dnf, which is, for example, a pointof the farthest focus of the narrow-angle lens 33 n.

In particular, the far point Dwf of the wide-angle camera 32 w is set tobe farther from a driver of the vehicle V than the close point Dnc ofthe narrow-angle camera 32 n therefrom, and the close point Dnc of thenarrow-angle camera 32 n is set to be farther from a drive of thevehicle V than the close point Dwc of the wide-angle camera 32 wtherefrom. Additionally, the far point Dnf of the narrow-angle camera 32n is set to be farther from a driver of the vehicle V than the far pointDwf of the wide-angle camera 32 w therefrom.

These settings enable

(1) The far point Dwf of the wide-angle camera 32 w to be locatedbetween the close and far points Dnc and Dnf of the narrow-angle camera32 n

(2) An overlap region Rnw, in which the depth of field Dn of thenarrow-angle camera 32 n and the depth of field Dw of the wide-anglecamera 32 w overlap with each other in the normal directionperpendicular to the corresponding angular fields of view, to beestablished

The lens barrel 34 t of the telephoto camera 32 t is positioned in thefirst casing segment 41 such that at least a predetermined rearprincipal point of the telephoto lens 33 t is vertically andhorizontally aligned with a corresponding predetermined rear principalpoint of the narrow-angle lens 33 n. Additionally, the optical axis Atof the telephoto camera 32 t is eccentrically adjusted with respectiveto each of the optical axis Aw of the wide-angle camera 32 w and theoptical axis An of the narrow-angle camera 32 n in the verticaldirection to thereby maintain the horizontal position of the opticalaxis At of the telephoto camera 32 t vertically aligning with thehorizontal position of each of the optical axis Aw and the optical axisAn.

Because the telephoto camera 32 t uses the telephoto lens 33 t, theangular field of view θt of the lens 33 t of the telephoto camera 32 tis set to a small angle narrower than each of the angular field of viewθw of the lens 33 w and the angular field of view θn of the lens 33 n,such as, for example, 35 degrees. These settings enable

(1) The angular field of view θt of the lens 33 t of the telephotocamera 32 t to partly overlap with the angular field of view θn of thelens 33 n of the narrow-angle camera 32 n in the normal directionperpendicular to the corresponding angular fields of view

(2) The angular field of view θt of the lens 33 t of the telephotocamera 32 t to partly overlap with the angular field of view θw of thelens 33 w of the wide-angle camera 32 w in the normal directionperpendicular to the corresponding angular fields of view

The telephoto camera 32 t has a depth of field Dt within the angularfield of view θt of the telephoto lens 33 t; the depth of field Dt isset to be within the range from a predetermined close point Dtc, whichis, for example, a point of the closest focus of the telephoto lens 33t, and a predetermined far point Dtf, which is, for example, a point ofthe farthest focus of the telephoto lens 33 t.

In particular, the far point Dnf of the narrow-angle camera 32 n is setto be farther from a driver of the vehicle V than the close point Dtc ofthe telephoto camera 32 t, and the close point Dtc of the telephotocamera 32 t is set to be farther from a driver of the vehicle V than

(1) The close point Dnc of the narrow-angle camera 32 n therefrom

(2) Each of the close point Dwc and the far point Dwf of the wide-anglecamera 32 w therefrom

Additionally, the far point Dtf of the telephoto camera 32 t is set tobe farther from a driver of the vehicle V than

(1) The far point Dnf of the narrow-angle camera 32 n therefrom

(2) The far point Dwf of the wide-angle camera 32 w therefrom

These settings enable

(1) The far point Dnf of the narrow-angle camera 32 n to be locatedbetween the close and far points Dtc and Dtf of the telephoto camera 32t

(2) An overlap region Rtn, in which the depth of field Dt of thetelephoto camera 32 t and the depth of field Dn of the narrow-anglecamera 32 n overlap with each other in the normal direction of thecorresponding angular fields of view, to be established

In particular, the far point Dwf of the telephoto camera 32 w isarranged to be outside a region defined between the close and far pointsDtc and Dtf of the telephoto camera 32 t, so that the depth of field Dtof the telephoto camera 32 t and the depth of field Dw of the wide-anglecamera 32 w deviate from each other. This results in the depth of fieldDt of the telephoto camera 32 t and the depth of field Dw of thewide-angle camera 32 w having non-overlap with each other in the normaldirection of the corresponding angular fields of view.

As described above, the wide-angle camera 32 w, the narrow-angle camera32 n, and the telephoto camera 32 t are arranged in the camera module 30such that the horizontal positions of the rear principle points of therespective cameras 32 w, 32 n, and 32 t are aligned with each other inthe vertical direction, i.e. the height direction, of the vehicle V.

The wide-angle camera 32 w, the narrow-angle camera 32 n, and thetelephoto camera 32 t, which are configured to include the respectivewide-angle lens 33 w, narrow-angle lens 33 n, and telephoto lens 33 t,enable the angular fields of view θw, θn, and θt to be different fromeach other while partly overlapping with each other in the normaldirection perpendicular to the corresponding angular fields of view.

The depths of field within the respective angular fields of eachadjacent pair of the cameras 32 w, 32 n, and 32 t are configured topartly overlap with each other in the normal direction perpendicular tothe corresponding angular fields of view.

Note that the exemplary embodiment defines the imaging region of thewide-angle camera 32 w, which is referred to as a wide-angle imagingregion, as a combination of the angular field of view θw and thecorresponding depth of field Dw, and also defines the imaging region ofthe narrow-angle camera 32 n, which is referred to as a narrow-angleimaging region, as a combination of the angular field of view θn and thecorresponding depth of field Dn. In addition, the exemplary embodimentdefines the imaging region of the telephoto camera 32 t, which isreferred to as a telephoto imaging region, as a combination of theangular field of view θt and the corresponding depth of field Dt.

These definitions therefore result in the wide-angle imaging region ofthe wide-angle camera 32 w, the narrow-angle imaging region of thenarrow-angle camera 32 n, and the telephoto imaging region of thetelephoto camera 32 t being different from each other while partlyoverlapping with each other in the normal direction perpendicular to thecorresponding angular fields of view.

In particular, because the wide-angle imaging region, the telephotoimaging region, and the narrow-angle imaging region, respectively covera relatively closer region around the vehicle V, a relatively fartherregion around the vehicle V, and a middle region around the vehicle Vbetween the relatively closer region and the relatively farther region.

Partial overlap among the wide-angle imaging region, the narrow-angleimaging region, and the telephoto imaging region results in frameimages, which are captured by the respective cameras 32 w, 32 n, and 32t based on their imaging regions, including a common region thereamong.

The control circuit 46 is configured to perform a known alignment taskbased on

(1) A first reference image of the wide-angle imaging region

(2) A second reference image of the narrow-angle imaging region (3) Athird reference image of the telephoto imaging region

The known alignment task enables misalignment among the positions, i.e.positional coordinates, of the respective optical axes Aw, An, and At tobe corrected based on the first to third reference images.

That is, the ECU 10 is configured to use two-dimensional frame images,i.e. closer images, of the wide-angle imaging region captured by thecamera 32 w, two-dimensional images, i.e. middle images, of thenarrow-angle imaging region captured by the camera 32 n, andtwo-dimensional images, i.e. farther images, of the telephoto imagingregion captured by the camera 32 t to thereby recognize objects existingaround the vehicle V. That is, the ECU 10 enables objects existing overa wide range from the relatively closer region around the vehicle V tothe relatively farther region around the vehicle V to be easilyrecognized.

Additionally, because the wide-angle imaging region, the narrow-angleimaging region, and the telephoto imaging region partly overlap witheach other, the closer images of the wide-angle imaging region, themiddle images of the narrow-angle imaging region, and the farther imagesof the telephoto imaging region partly overlap with other. Thistherefore prevents misrecognition of an object existing in at least oneof overlap areas among the first to third images even if the object ismoving the at least one of the overlap areas, thus recognizing theobject with higher accuracy.

As described above, the camera module 30, which is comprised of thethree cameras 32 w, 32 n, and 32 t, enables the ECU 10 to recognize,with higher accuracy, lane markers on a scheduled road of the vehicle Vand/or objects existing in the surrounding region around the vehicle V.

However, if at least one of the cameras 32 w, 32 n, and 32 t of thecamera module 30 has malfunctioned, the recognition accuracy of lanemarkers and/or objects in a corresponding one of the relatively closer,farther, and middle regions around the vehicle V may decrease. If theECU 10 continuously carried out autonomous driving of the vehicle V incooperation with the other ECUs 24, 26, and 28 with decreasingrecognition accuracy in one of the relatively closer, farther, andmiddle regions around the vehicle V, it would be difficult to performautonomous driving of the vehicle V accordingly, i.e. reliably andsafely.

From this viewpoint, the camera monitor unit 8 of the ECU 10 isconfigured to monitor each of the cameras 32 w, 32 n, and 32 t tothereby determine whether each of the cameras 32 w, 32 n, and 32 t isoperating normally during autonomous driving of the vehicle V. Thecamera monitor unit 8 is also configured to

(1) Discontinue the autonomous driving of the vehicle V upon determiningthat the camera 32 n has malfunctioned

(2) Limit, upon determining that one of the cameras 32 t and 32 w hasmalfunctioned, the autonomous driving conditions set by the drivecontrol unit 6 in accordance with the imaging area of the malfunctioningcamera while continuing the autonomous driving of the vehicle V

As described above, the CPU 2 executes instructions of the at least onecontrol program stored in the memory 4 to thereby implement thefunctions of the camera monitor unit 8. In other words, the CPU 2executes instructions of the at least one control program to therebyserve as the camera monitor unit 8 to execute a camera monitor routineillustrated as a flowchart in FIG. 5 every predetermined control period.

When starting the camera monitor routine as a main routine, the CPU 2determines whether the ECUs 24, 26, and 28 is carrying out theautonomous driving of the vehicle V based on the autonomous drivingconditions set thereby in step S110. The CPU 2 terminates the cameramonitor routine upon determining that the ECUs 24, 26, and 28 is notcarrying out the autonomous driving of the vehicle V (NO in step S110).Otherwise, upon determining that the ECUs 24, 26, and 28 is carrying outthe autonomous driving of the vehicle V (YES in step S110), the CPU 2sends, to the control circuit 46 of the camera module 30, acommunication request for each of the cameras 32 w, 32 n, and 32 t, andreceives, from each of the cameras 32 w, 32 n, and 32 t via the controlcircuit 46, a response signal indicative of the operating state of thecorresponding one of the cameras 32 w, 32 n, and 32 t to thereby checkwhether the corresponding one of the cameras 32 w, 32 n, and 32 t isoperating in step S120.

In addition, upon determining that the ECUs 24, 26, and 28 is carryingout the autonomous driving of the vehicle V (YES in step S110), the CPU2 obtains, from each of the cameras 32 w, 32 n, and 32 t via the controlcircuit 46, an image data item, i.e. a frame image captured by thecorresponding one of the cameras 32 w, 32 n, and 32 t to thereby checkwhether the image data item obtained from the corresponding one of thecameras 32 w, 32 n, and 32 t is a normally captured image data item instep S130.

Note that the CPU 2 can carry out the operation in step S120 and theoperation in step S130 in random order or carry out the operation instep S120 and the operation in step S130 in parallel.

Additionally, the CPU 2 can directly receive the response signal fromeach of the cameras 32 w, 32 n, and 32 t, and/or directly receive theimage data item from each of the cameras 32 w, 32 n, and 32 t.

After the operation check of each camera 32 w, 32 n, 32 t in step S120and the image-data check of each camera 32 w, 32 n, 32 t in step S130are completed, the CPU 2 determines, based on the checked results insteps S120 and S130, whether the narrow-angle camera 32 n hasmalfunctioned in step S140.

Specifically, the CPU 2 determines that the narrow-angle camera 32 n hasmalfunctioned upon determining that the checked result in step S120represents that the camera 32 n is not operating due to, for example, noresponse signal is returned from the camera 32 n. In addition, the CPU 2deter mines that the narrow-angle camera 32 n has malfunctioned upondetermining that the checked result in step S130 represents that theimage data item obtained from the camera 32 n is an abnormally capturedimage due to, for example, overexposure of the camera 32 n,underexposure of the camera 32 w, damages of the lens 33 n, and/or dutyon the lens 33 n. Note that the CPU 2 can determine that the image dataitem obtained from the camera 32 w is an abnormally captured image basedon, for example, a pixel value, i.e. a luminance value, of each pixel ofthe imaging device 35 n representing the intensity of the correspondingreceived light component.

Upon determining that the camera 32 n has malfunctioned (YES in stepS140), the CPU 2 visibly and/or audibly outputs, to a driver of thevehicle V through the output unit 17, a message indicative of the camera32 w having malfunctioned, thus urging a driver of the vehicle V todiscontinue the autonomous driving of the vehicle V in step S150.

That is, the operation in step S150 prompts a driver of the vehicle V toinstruct the ECU 10 to switch the operation mode of the vehicle V fromthe autonomous driving mode to the manual driving mode using the inputunit 18, thus discontinuing the autonomous driving of the vehicle V.

Alternatively, the CPU 2 can perform an emergency braking control taskto thereby forcibly stop the vehicle V in step S150 for example if adriver of the vehicle V does not start driving of the vehicle V in themanual driving mode. For example, the emergency braking control task isconfigured to cause the ECUs 22, 24, and 26 to safely pull the vehicle Vover to a safe place while outputting brake power to thereby deceleratethe vehicle V, thus autonomously parking the vehicle V at the safeplace.

That is, the exemplary embodiment disables the drive control unit 6 fromexecuting the autonomous driving control task, thus discontinuing theautonomous driving of the vehicle V upon it being determined that thereis a malfunction or fault in the narrow-angle camera 3 nw so that it isdifficult to obtain frame images on the middle region around the vehicleV, which is needed for autonomous driving of the vehicle V. Thistherefore ensures the safety of the vehicle V.

Otherwise, upon determining that the camera 32 n has not malfunctioned(NO in step S140), the CPU 2 determines, based on the checked results insteps S120 and S130, whether the telephoto camera 32 t has malfunctionedin step S160.

Specifically, the CPU 2 determines that the telephoto camera 32 t hasmalfunctioned upon determining that the checked result in step S120represents that the camera 32 t is not operating due to, for example, noresponse signal is returned from the camera 32 t. In addition, the CPU 2determines that the telephoto camera 32 t has malfunctioned upondetermining that the checked result in step S130 represents that theimage data item obtained from the camera 32 t is an abnormally capturedimage due to, for example, overexposure of the camera 32 t,underexposure of the camera 32 t, damage to the lens 33 t, and/or dirton the lens 33 t. Note that the CPU 2 can determine that the image dataitem obtained from the camera 32 t is an abnormally captured image basedon, for example, a pixel value of each pixel of the imaging device 35 trepresenting the intensity of the corresponding received lightcomponent.

Upon determining that the camera 32 t has malfunctioned (YES in stepS160), the CPU 2 reduces the speed limit in the autonomous drivingconditions to be lower by a predetermined speed to accordingly limit thespeed of the vehicle V during the autonomous driving of the vehicle V,i.e. limit, i.e. tighten, the autonomous driving conditions, in stepS170.

Additionally, the CPU 2 visibly and/or audibly outputs, to a driver ofthe vehicle V through the output unit 17, a message indicative of atleast one of information about the upper speed limit of the vehicle Vbeing limited or the changed upper speed limit in step S170. Inaddition, the CPU 2 can visibly and/or audibly output, to a driver ofthe vehicle V through the output unit 17, a message indicative of thecamera 32 t having malfunctioned in step S170.

The reason why to reduce the speed limit upon the telephoto camera 32 thaving malfunctioned is that the malfunction of the telephoto camera 32t may make it difficult to recognize, from the closer and middle imagescaptured by the wide- and narrow-angle cameras 32 w and 32 n, objectsexisting in the relatively farther region covered by the telephotocamera 32 t.

That is, even if objects existing in the relatively closer region andthe middle region are recognized based on the closer and middle images,an increase of the speed of the vehicle V may cause a delay inrecognition of objects existing in the relatively farther region,causing sudden braking and/or sudden steering of the vehicle V.

From this viewpoint, restricting the speed limit in step S170 preventstravelling of the vehicle V in the autonomous driving mode from becomingunstable.

Otherwise, upon determining that the camera 32 t has not malfunctioned(NO in step S160), the CPU 2 determines, based on the checked results insteps S120 and S130, whether the wide-angle camera 32 w hasmalfunctioned in step S180.

Specifically, the CPU 2 determines that the wide-angle camera 32 w hasmalfunctioned upon determining that the checked result in step S120represents that the camera 32 w is not operating due to, for example, noresponse signal is returned from the camera 32 w. In addition, the CPU 2determines that the wide-angle camera 32 w has malfunctioned upondetermining that the checked result in step S130 represents that theimage data item obtained from the camera 32 w is an abnormally capturedimage due to, for example, overexposure of the camera 32 w,underexposure of the camera 32 w, damages of the lens 33 w, and/or dutyon the lens 33 w. Note that the CPU 2 can determine that the image dataitem obtained from the camera 32 w is an abnormally captured image basedon, for example, a pixel value of each pixel of the imaging device 35 wrepresenting the intensity of the corresponding received lightcomponent.

Upon determining that the camera 32 w has malfunctioned (YES in stepS180), the CPU 2 disables autonomous steering of the vehicle V in eachtraffic intersection in the scheduled road to accordingly limit, i.e.tighten, an autonomous-driving execution condition included in theautonomous driving conditions in step S190.

Additionally, the CPU 2 visibly and/or audibly outputs, to a driver ofthe vehicle V through the output unit 17, a message indicative of atleast one of information about inexecution of autonomous steering ineach traffic intersection in step S190. In addition, the CPU 2 canvisibly and/or audibly outputs, to a driver of the vehicle V through theoutput unit 17, a message indicative of the camera 32 w havingmalfunctioned in step S190.

That is, the malfunction of the wide-angle camera 32 w may make itdifficult to recognize, from the farther and middle images captured bythe wide- and narrow-angle cameras 32 t and 32 n, objects existing inthe relatively closer region covered by the wide-angle camera 32 w. Thistherefore would reduce the safety of the autonomous driving of thevehicle V if an autonomous left turn or an autonomous right turn of thevehicle V based on the autonomous driving of the vehicle V were carriedout in a traffic intersection while there is a malfunction in thewide-angle camera 32 w.

From this viewpoint, disabling autonomous steering of the vehicle V ineach traffic intersection and informing a driver of the vehicle V aboutprohibition of autonomous-steering in each traffic intersection in stepS190 enable a driver of the vehicle V to drive the vehicle V in themanual driving mode, thus ensuring the safety of the vehicle V.

Note that the CPU 2 can limit, i.e. tighten, the autonomous drivingconditions to thereby permit the vehicle V to travel straight in eachtraffic intersection while setting the speed of the vehicle V to belower than the speed limit set in step S19 for example if a driver ofthe vehicle V does not start driving of the vehicle V in the manualdriving mode.

Otherwise, upon determining that the camera 32 w has not malfunctioned(NO in step S180), the CPU 2 serves as the drive control unit 6 tocontinuously carry out the autonomous driving of the vehicle V in stepS200, thus terminating a current cycle of the camera monitor routine.

Note that the CPU 2 can carry out the operation in step S140 and theoperation in step S160 in random order.

As described above, the camera monitor unit 8 of the ECU 10 of theexemplary embodiment is configured to

(1) Check whether at least one of the wide-angle camera 32 w, thenarrow-angle camera 32 n, and the telephoto camera 32 t hasmalfunctioned

(2) Limit the speed of the vehicle V as one of the autonomous drivingconditions required for autonomous driving of the vehicle V upon itbeing determined that the telephoto camera 32 t has malfunctioned tothereby continuously carry out the autonomous driving of the vehicle V

(3) Disable autonomous steering of the vehicle V in each trafficintersection as one of the autonomous driving conditions required forautonomous driving of the vehicle V upon it being determined that thewide-angle camera 32 n has malfunctioned to thereby limit autonomousdriving tasks executable in each traffic intersection while continuouslycarrying out the autonomous driving of the vehicle V

This therefore enables the drive control unit 6 of the ECU 10 tocontinuously carry out autonomous driving of the vehicle V whilelimiting the autonomous driving conditions even if it is determined bythe camera monitor unit 8 that the telephoto camera 32 t or thewide-angle camera 32 w has malfunctioned.

Consequently, the ECU 10 of the exemplary embodiment is configured toeliminate the need to discontinue, i.e. cancel, the autonomous drivingof the vehicle V, and switch the operation mode of the vehicle V fromthe autonomous driving mode to the manual driving mode even if at leastone of the telephoto camera 32 t or the wide-angle camera 32 w hasmalfunctioned. That is, this configuration of the ECU 10 enablesautonomous driving of the vehicle V to be continuously carried out basedon the limited autonomous driving conditions as long as at least thenarrow-angle camera 32 n is operating normally. This therefore producesan autonomous driving system 1 being more convenient for driving of thevehicle V.

The present disclosure is not limited to the above exemplary embodimentset forth above, and can be variously modified for example as follows.

The camera module 30 of the exemplary embodiment is comprised of thethree cameras, i.e. the wide-angle camera 32 w, narrow-angle camera 32n, and telephoto camera 32 t, but the present disclosure is not limitedthereto.

Specifically, the camera module 30 can be comprised of a first cameraincluding a normal lens or a standard lens, and a second cameraincluding a telephoto lens whose angular field of view is narrower thanan angular field of view of the normal lens. Note that the normal orstandard lens is defined as a lens that has a focal length equal orclose to the diagonal of the light receiving area of a correspondingimaging device.

Additionally, the camera module 30 can be comprised of four cameras thatrespectively capture an upper part, a lower part, a right part, and aleft part of a front view, i.e. front imaging region, of the vehicle V.

That is, the camera module 30 can include plural cameras withrespectively different imaging regions from each other around thevehicle V, and can be configured such that images captured from therespective different imaging regions enable one or more objects existingin at least one of the imaging regions to be recognized. At that time,if some cameras have malfunctioned, the camera monitor unit 6 can beconfigured to limit the autonomous driving conditions. This enables thedrive control unit 6 to continuously carry out the autonomous driving ofthe vehicle V.

At least one of the autonomous driving conditions, which is limited bythe ECU 10, can be selected by the ECU 10 depending on which of theplural cameras has malfunctioned.

For example, let the camera module 30 be comprised of four cameras thatrespectively capture an upper part, a lower part, a right part, and aleft part of a front view, i.e. front imaging region, of the vehicle Vin the travelling direction. In this case, if the camera for capturingthe right part of the front view has malfunctioned, the ECU 10 can beconfigured to limit changing of a currently travelling lane of thevehicle V to an adjacent right-side lane of the currently travellinglane, and/or right turn of the vehicle V.

The plural imaging regions of plural cameras can be configured to atleast partly overlap with each other as described in the exemplaryembodiment. This configuration prevents the ECU 10 from missing anobject even if the object has moved over at least one of boundariesbetween the overlapping imaging regions.

The cameras 32 w, 32 n, and 32 t, which are installed in the cameracasing 40, can be directly mounted to the vehicle V.

The CPU 2 of the ECU 10 of the exemplary embodiment executes the atleast one control program to thereby implement functions as the drivecontrol unit 6 and functions as the camera monitor unit 8.

In contrast, the control circuit 46 of the camera module 30 can beconfigured to implement the functions as the functions as the cameramonitor unit 8, and the ECU 10 can be configured to implement thefunctions as the drive control unit 6. In this modification, if thecontrol circuit 46 of the camera module 30 is comprised of amicrocomputer, the microcomputer of the control circuit can execute atleast one control program stored therein to thereby execute the cameramonitor routine illustrated in FIG. 5 corresponding to the at least onecontrol program as the functions of the camera monitor unit 8. That is,the control circuit 46 of the camera module 30 can serve as, forexample, the camera monitor unit 8.

That is, at least part of all functions provided by the ECU 10 can beimplemented by at least one processor; the at least one processor can becomprised of

(1) The combination of at least one programmed processing unit, i.e. atleast one programmed logic circuit, and at least one memory includingsoftware that causes the at least one programed logic circuit toimplement all the functions

(2) At least one hardwired logic circuit that implements all thefunctions

(3) At least one hardwired-logic and programmed-logic hybrid circuitthat implements all the functions

While the illustrative embodiment of the present disclosure has beendescribed herein, the present disclosure is not limited to theembodiment and its modifications described herein, but includes any andall embodiments having modifications, omissions, combinations (e.g., ofaspects across various embodiments), adaptations and/or alterations aswould be appreciated by those in the art based on the present disclosurewithin the scope of the present disclosure.

For example, each of the technical features described in the embodimentand its modifications can be replaced with a known structure having thesame function as the corresponding technical feature. Each of thetechnical features described in the embodiment and its modifications canalso be combined with at least one of the other technical features. Atleast one of the technical features described in the embodiment and itsmodifications can further be eliminated unless the at least one of thetechnical features is described as an essential element in the presentspecification.

The functions of the drive control unit and the camera monitor unit canbe implemented by various embodiments; the various embodiments includeautonomous driving control ECUs, camera modules, programs for serving acomputer as the functions, storage media, such as non-transitory media,storing the programs, and autonomous driving control methods.

What is claimed is:
 1. An autonomous driving control apparatusinstallable in a vehicle that includes: at least first and secondcameras configured to respectively capture at least first and secondimages of at least first and second directional regions that are atleast partly different from each other from the vehicle; and a vehiclecontrol unit configured to control a travelling state of the vehicle,the autonomous driving control apparatus comprising: a drive controlunit configured to: determine, based on the first and second images andan autonomous driving condition, a value of at least one controlledvariable for autonomous driving of the vehicle; and output, to thevehicle control unit, the value of the at least one controlled variableto thereby cause the vehicle control unit to carry out a task ofautonomously driving the vehicle; and a camera monitor unit configuredto: determine whether one of the first and second cameras hasmalfunctioned; and limit, when determining that one of the first andsecond cameras has malfunctioned, the autonomous driving condition basedon one of the first and second directional regions corresponding to oneof the first and second cameras having malfunctioned, the drive controlunit being configured to determine the value of the at least onecontrolled variable in accordance with the limited autonomous drivingcondition to thereby cause the vehicle control unit to continuouslycarry out the task of autonomously driving the vehicle in accordancewith the determined value of the at least one controlled variable. 2.The autonomous driving control apparatus according to claim 1, wherein:the first camera is a telephoto camera whose first directional region isfarther from the vehicle than the second directional region of thesecond camera therefrom; the autonomous driving condition includes aspeed of the vehicle; and the camera monitor unit is configured to limitthe speed of the vehicle when determining that the first camera hasmalfunctioned.
 3. The autonomous driving control apparatus according toclaim 1, wherein: the first camera has a predetermined first angularfield of view; the second camera is a wide-angle camera having apredetermined second angular field of view that is wider than the firstangular field of view; the autonomous driving condition includes anexecution condition of an autonomous steering of the vehicle; and thecamera monitor unit is configured to limit the execution condition ofthe autonomous steering of the vehicle when determining that the secondcamera has malfunctioned.
 4. The autonomous driving control apparatusaccording to claim 1, wherein: the at least first and second camerascomprise at least first, second, and third cameras; the first camera isa telephoto camera having a predetermined first angular field of view,the first directional region of the first camera being farther from thevehicle than the second directional region of the second cameratherefrom; the second camera is a wide-angle camera having apredetermined second angular field of view that is wider than the firstangular field of view; the third camera is a narrow-angle camera having:a predetermined third directional region closer to the vehicle than thefirst directional region thereto and farther from the vehicle than thesecond directional region therefrom; and a predetermined third angularfield of view wider than the first angular field of view and narrowerthan the second angular field of view; and the camera monitor unit isconfigured to perform at least one of: a first task of prompting adriver of the vehicle to discontinue the task of autonomously drivingthe vehicle when determining that the third camera has malfunctioned;and a second task of forcibly discontinuing the task of autonomouslydriving the vehicle when determining that the third camera hasmalfunctioned.
 5. The autonomous driving control apparatus according toclaim 4, wherein: the first, second, and third cameras are aligned witheach other in a height direction of the vehicle.
 6. The autonomousdriving control apparatus according to claim 4, wherein: the first,second, and third directional regions are arranged to at least partlyoverlap with each other in a height direction of the vehicle.
 7. Aprogram product for a vehicle that includes: at least first and secondcameras configured to respectively capture at least first and secondimages of at least first and second directional regions that are atleast partly different from each other from the vehicle; a vehiclecontrol unit configured to control a travelling state of the vehicle;and a drive control unit configured to: determine, based on the firstand second images and an autonomous driving condition, a value of atleast one controlled variable for autonomous driving of the vehicle; andoutput, to the vehicle control unit, the value of the at least onecontrolled variable to thereby cause the vehicle control unit to carryout a task of autonomously driving the vehicle, the computer programproduct comprising: a non-transitory computer-readable storage medium;and a set of computer program instructions stored in thecomputer-readable storage medium, the instructions causing a computer tocarry out: a first step of determining whether one of the first andsecond cameras has malfunctioned; and a second step of limiting, when itbeing determined one of the first and second cameras has malfunctioned,the autonomous driving condition based on one of the first and seconddirectional regions corresponding to the one of the first and secondcameras having malfunctioned.